Container scheduling method and apparatus, and non-volatile computer-readable storage medium

ABSTRACT

A container scheduling method and apparatus, and a computer-readable storage medium, which relate to the technical field of computers. The method includes: according to a resource usage amount of a container set copy which has run, determining a predicted resource usage amount of a container set copy to be scheduled, wherein the type of container set copy which has run is the same as the type of container set copy to be scheduled; according to the predicted resource usage amount and a resource supply amount supported by each candidate node, determining a candidate node matching the container set copy which has run; and scheduling the container set copy which has run to the matched candidate node for running.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims the priority to the Chinesepatent application No. 201911175494.6 filed on Nov. 26, 2019, thedisclosure of which is hereby incorporated as a whole into the presentapplication.

TECHNICAL FIELD

This disclosure relates to the field of computer technologies, andparticularly, to a scheduling method and apparatus of containers, and anon-transitory computer-readable storage medium.

BACKGROUND

A container cloud is an important type of a current cloud computingplatform, and is mainly applied in a private cloud scene to meet demandsof customers for privatization and customization. When the containercloud is established by the customers, a container cloud cluster will bedeployed in a self-built machine room, where hundreds of physicalservers will be often deployed to carry business on the container cloud.

For example, a plurality of nodes can be configured as hosts for therunning of containers. The nodes can be physical machines or virtualmachines depending on the physical machines. Therefore, it is necessaryto generate a suitable scheduling strategy for scheduling the containersto run on corresponding nodes.

In the related art, the scheduling strategy is generated mainlyaccording to how resources are distributed evenly on each node.

SUMMARY

According to some embodiments of the present disclosure, there isprovided a scheduling method of containers, comprising: determiningpredicted resource usage of a to-be-scheduled Pod replica according toresource usage of an already run Pod replica, wherein types of thealready run Pod replica and the to-be-scheduled Pod replica are thesame; determining at least one candidate node matched with the alreadyrun Pod replica according to the predicted resource usage and resourceprovision supported by candidate nodes; and scheduling the already runPod replica to be run on the at least one candidate node.

In some embodiments, the determining at least one candidate node matchedwith the already run Pod replica comprises: determining the at least onecandidate node matched with the already run Pod replica according towhether predicted resource usage of each type of resource in thepredicted resource usage is matched with provision of each type ofresource in the resource provision.

In some embodiments, the determining at least one candidate node matchedthe already run Pod replica comprises: judging whether the predictedresource usage of each type of resource in the predicted resource usageis greater than a usage threshold corresponding to each type ofresource; determining a resource with predicted resource usage greaterthan a corresponding usage threshold as a demand resource; judgingwhether a current utilization rate, in each of the at least onecandidate node, of each type of resource in the resource provision isless than a utilization rate threshold corresponding to each type ofresource; determining a resource with a current utilization rate lessthan a corresponding utilization rate threshold as an available resourceof each of the at least one candidate node; and determining the at leastone candidate node matched with the already run Pod replica according tothe matching of the demand resource and the available resource.

In some embodiments, the scheduling the already run Pod replica to berun on the at least one candidate node comprises: determining a numberof physical machines that need to be awakened by the at least onecandidate node according to the predicted resource usage and theresource provision of the at least one candidate node, wherein thenumber of the physical machines is a minimum number capable of meetingthe predicted resource usage.

In some embodiments, the method further comprises: acquiring ahistorical change of the resource usage of the already run Pod replica;judging whether the to-be-scheduled Pod replica is currently in a peakor valley time period of a usage according to the historical change; anddetermining whether to increase a number of the to-be-scheduled Podreplica according to a result of the judgment.

In some embodiments, the acquiring a historical change of the resourceusage of the already run Pod replica comprises: calculating averageusage of each type of resource within a specified time period accordingto the resource usage of the already run Pod replica at each time momentwithin the specified time period; and determining the peak and valleytime periods of the usage of each type of resource within the specifiedtime period as the historical change according to the average usage.

In some embodiments, the method further comprises: estimating resourceusage of the to-be-scheduled Pod replica at a target time momentaccording to the historical change; and under the condition that thecurrently awakened physical machine is incapable of meeting the resourceusage at the target time moment, awakening a number of physical machineswhich are capable of meeting the resource usage at the target timemoment to provide resources before the target time moment comes.

In some embodiments, the determining whether to increase a number of theto-be-scheduled Pod replica according to a result of the judgmentcomprises: under the condition that it is in the peak time period and aresource load of a certain to-be-scheduled Pod replica exceeds a loadthreshold, increasing the number of the to-be-scheduled Pod replica.

In some embodiments, the determining whether to increase a number of theto-be-scheduled Pod replica according to a result of the judgmentcomprises: under the condition that it is in the valley time period,decreasing the number of the to-be-scheduled Pod replica, andcontrolling an cleared physical machine to enter a sleep state.

In some embodiments, the method further comprises: according to thenumber of Pod replicas running on a corresponding physical machine ofthe at least one candidate node, controlling the corresponding physicalmachine to be in a high or non-high performance mode.

In some embodiments, the controlling the corresponding physical machineto be in a high or non-high performance mode comprises: under thecondition that there is no Pod replica running on the correspondingphysical machine, controlling the corresponding physical machine to bein a sleep mode; and under the condition that the number of Pod replicasrunning on the corresponding physical machine is less than a replicathreshold, controlling the corresponding physical machine to be in a lowperformance mode.

In some embodiments, each of the at least one candidate node isdetermined according to at least one of the number of theto-be-scheduled Pod replica, a high availability demand, an affinitydemand, or an exclusivity demand.

According to other embodiments of the present disclosure, there isprovided a scheduling apparatus of containers, comprising: a usagedetermination unit configured to determine predicted resource usage of ato-be-scheduled Pod replica according to resource usage of an alreadyrun Pod replica, wherein types of the already run Pod replica and theto-be-scheduled Pod replica are the same; a node matching unitconfigured to determine at least one candidate node matched with thealready run Pod replica according to the predicted resource usage andresource provision supported by candidate nodes; and a scheduling unitconfigured to schedule the already run Pod replica to be run on the atleast one candidate node.

In some embodiments, the node matching unit determines the at least onecandidate node matched with the already run Pod replica according towhether predicted resource usage of each type of resource in thepredicted resource usage is matched with provision of each type ofresource in the resource provision.

In some embodiments, the node matching unit judges whether the predictedresource usage of each type of resource in the predicted resource usagesis greater than a usage threshold corresponding to each type ofresource; determines a resource greater than the corresponding usagethreshold as a demand resource; judges whether a current utilizationrate, in each of the at least one candidate node, of each type ofresource in the resource provision is less than a correspondingutilization rate threshold; determines a resource with a currentutilization rate less than a corresponding utilization rate threshold asan available resource of each of the at least one candidate node; anddetermines the candidate nodes matched with the already run Pod replicaaccording to the matching of the demand resource and the availableresource.

In some embodiments, the scheduling unit determines the number ofphysical machines that need to be awakened by the at least one candidatenode according to the predicted resource usage and the resourceprovision of the at least one candidate node, wherein the number ofphysical machines is a minimum number capable of meeting the predictedresource usage.

In some embodiments, the apparatus further comprises a judgment unitconfigured to judge whether the to-be-scheduled Pod replica is currentlyin a peak or valley time period of a usage according to an acquiredhistorical change of the resource usage of the already run Pod replica,so as to determine whether to increase the number of the to-be-scheduledPod replica according to the judgment result.

In some embodiments, the judgment unit calculates average usage of eachtype of resource within a specified time period according to theresource usage of the already run Pod replica at each time moment withinthe specified time period; and according to the average usage,determines the peak and valley time periods of the usage of each type ofresource within the specified time period as the historical change.

In some embodiments, the apparatus further comprises a control unitconfigured to estimate resource usage of the to-be-scheduled Pod replicaat a target time moment according to the historical change; and underthe condition that the currently awakened physical machine is incapableof meeting the resource usage at the target time moment, awaken a numberof physical machines which are capable of meeting the resource usage atthe target time moment to provide resources before the target timemoment comes.

In some embodiments, the control unit, under the condition that it is inthe peak time period and a resource load of a certain to-be-scheduledPod replica exceeds a load threshold, increases the number of theto-be-scheduled Pod replica.

In some embodiments, the control unit, under the condition that it is inthe valley time period, reduces the number of to-be-scheduled Podreplica and controls an cleared physical machine to enter a sleep state.

In some embodiments, the control unit is configured to control,according to the number of Pod replicas running on a correspondingphysical machine of the at least one candidate node, the correspondingphysical machine to be in a high or non-high performance mode.

In some embodiments, the control unit controlling the correspondingphysical machine to be in a high or non-high performance mode comprises:under the condition that there is no Pod replica running on thecorresponding physical machine, controlling the corresponding physicalmachine to be in a sleep mode; and under the condition that the numberof Pod replicas running on the corresponding physical machine is lessthan a replica threshold, controlling the corresponding physical machineto be in a low performance mode.

In some embodiments, each of the at least one candidate node isdetermined according to at least one of the number of theto-be-scheduled Pod replica, a high availability demand, an affinitydemand, or an exclusivity demand.

According to still other embodiments of the present disclosure, there isprovided a scheduling apparatus of containers, comprising: a memory; anda processor coupled to the memory, the processor being configured toperform, based on instructions stored in the memory, the schedulingmethod of the containers in any of the above embodiments.

According to further embodiments of the present disclosure, there isprovided a non-transitory computer-readable storage medium havingthereon stored a computer program which, when executed by a processor,implements the scheduling method of containers in any of the aboveembodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings described herein are used for providing afurther understanding of the present disclosure and constitute a part ofthis application, and illustrative embodiments of the present disclosureand their description are used for explaining the present disclosure anddo not constitute an improper limitation on the present disclosure. Inthe drawings:

FIG. 1 illustrates a flow diagram of some embodiments of a schedulingmethod of containers according to the present disclosure;

FIG. 2 illustrates a flow diagram of other embodiments of a schedulingmethod of containers according to the present disclosure;

FIG. 3 illustrates a schematic diagram of some embodiments of ascheduling apparatus of containers according to the present disclosure;

FIG. 4 a illustrates a schematic diagram of some embodiments of noderesource distribution according to the present disclosure;

FIG. 4 b illustrates a schematic diagram of some embodiments of resourceusage of a Pod replica according to the present disclosure;

FIG. 4 c illustrates a schematic diagram of some embodiments of resourceusage of two Pod replicas according to the present disclosure;

FIG. 4 d illustrates a schematic diagram of some embodiments of ascheduling method of containers according to the present disclosure;

FIG. 5 illustrates a schematic diagram of other embodiments of ascheduling method of containers according to the present disclosure;

FIG. 6 illustrates a block diagram of some embodiments of a schedulingapparatus of containers according to the present disclosure;

FIG. 7 illustrates a block diagram of other embodiments of a schedulingapparatus of containers according to the present disclosure;

FIG. 8 illustrates a block diagram of still other embodiments of ascheduling apparatus of containers according to the present disclosure.

DETAILED DESCRIPTION

The technical solutions in the embodiments of the present disclosurewill be clearly and completely described below in conjunction with theaccompanying drawings in the embodiments of the present disclosure, andit is obvious that the embodiments described are only some of theembodiments of the present disclosure, rather than all of them. Thefollowing description of at least one exemplary embodiment is merelyillustrative in nature and in no way serves as any limitation on thisdisclosure and its application or use. All other embodiments, which canbe derived by one of ordinary skill in the art from the embodiments inthe present disclosure without making creative efforts, fall within theprotection scope of the present disclosure.

The relative arrangements, numerical expressions and numerical values ofcomponents and steps set forth in these embodiments do not limit thescope of the present disclosure unless otherwise specified. Meanwhile,it should be understood that sizes of various portions shown in thedrawings are not drawn to actual scales for ease of description. Atechnique, method, and device known to one of ordinary skill in therelated art may not be discussed in detail but should be regarded aspart of the granted specification where appropriate. In all examplesshown and discussed herein, any specific value should be construed asexemplary only and not as limiting. Therefore, other examples of theexemplary embodiments can have different values. It should be notedthat: similar reference numbers and letters refer to similar items inthe following drawings, and therefore, once a certain item is defined inone drawing, it need not be discussed further in subsequent drawings.

Inventors of the present disclosure have found that, the related artdescribed above have the following problem: since a scheduling strategyis not generated from the perspective of cluster management andmaintenance, a mismatch between resource usage and resource provision iscaused, resulting in a waste of system resources.

In view of this, in the present disclosure, there is provided acontainer scheduling technical solution, capable of reducing the wasteof system resources.

As described above, when a customer establishes a container cloud, acontainer cloud cluster will be deployed in a self-built machine room,where hundreds of physical servers will often be deployed to carrybusiness on the container cloud. A large number of physical serversrunning for a long time will result in huge energy consumption, and themore physical servers run, the greater the scale required for matchedcooling and dust removal facilities is, which brings very hugecontinuous cost expenses such as electricity and management bills to thecustomer.

In view of the above technical problem, the present disclosure is basedon container cloud technologies, uses a mechanism that Kubernetes in acontainer cloud is capable of strategically scheduling a Pod(set ofcontainers), and focuses on clearing loads of part of physical machinesthrough the Pod scheduling, to enter a sleep state without energyconsumption. In this way, the resource usage can be matched with theresource demand as much as possible, which achieves the target of notonly meeting the business demand, but also saving energy and reducingcost. This can be achieved, for example, by the following embodiments.

FIG. 1 illustrates a flow diagram of some embodiments of a schedulingmethod of containers according to the present disclosure.

As shown in FIG. 1 , the method comprises: step 110, determiningpredicted resource usage; step 120, determining a matched candidatenode; and step 130, scheduling an already run Pod replica.

In the step 110, the predicted resource usage of a to-be-scheduled Podreplica is determined according to resource usage of the already run Podreplica. Types of the already run Pod replica and the to-be-scheduledPod replica are the same.

For example, Pod is a set of closely related containers and is a basicunit of Kubernetes scheduling. The plurality of containers in the Podshare a network and file system, so that they can complete a service insuch a simple and efficient manner of inter-process communication andfile sharing. A Pod replica is a set of Pods created by using one samePod mode, and is controlled by RC (Replication Controller) or RS(Replica Set).

In some embodiments, the type of the Pod replica is determined dependingon a corresponding Pod mode, and Pod replicas with the same Pod mode areof the same type. For example, the Pod mode can comprise at least one ofa mirror image of a container comprised in the Pod, a correspondingspecification of the mirror image, a corresponding service name, or thenumber of replicas. The specification can comprise hardware resourceconfiguration information required for the mirror image, such as thenumber of CPU (Central Processing Unit) cores, memory capacity, networkbandwidth, and storage capacity.

In some embodiments, one type of Pod mode can be determined by using onePod template file, for example, a structure of one type of Pod mode is:

{ Mirror image 1, specification 1; Mirror image 2, specification 2; ...Mirror image n, specification n; Service name; The number of replicas; }

The specification can comprise the hardware configuration informationsuch as the number of CPU cores, memory capacity, network bandwidth, andstorage capacity. For example, a Pod with a label of X is created in theabove manner, then the above structure is referred to as a Pod mode ofPod X.

In the step 120, the at least one candidate node matched with thealready run Pod replica is determined according to the predictedresource usage and resource provision supported by candidate nodes.

In some embodiments, each of the at least one candidate node isdetermined according to at least one of the number of theto-be-scheduled Pod replicas, a HA (High Availability) demand, anaffinity demand, or an exclusivity demand.

For example, affinity is a scheduling strategy when a Pod runs, andcomprises: node Affinity, pod Affinity, and pod Anti-Affinity. The nodeAffinity is used for specifying on which node a Pod can or cannot bedeployed. The pod Affinity is used for specifying along with which Podsthe Pod can be deployed under one same topology. The pod Anti-Affinityis used for specifying along with which Pods the Pod cannot be deployedunder the same topology, and together with the pod Affinity, settlesrelations between the pods.

In some embodiments, the at least one candidate node matched with thealready run Pod replica is determined according to whether predictedresource usage of each type of resource in the predicted resource usageis matched with provision of each type of resource in the resourceprovision.

In some embodiments, it is judged whether the predicted resource usageof each type of resource in the predicted resource usage is greater thana corresponding usage threshold; a type of resource greater than thecorresponding usage threshold is determined as a demand resource; it isjudged whether a current utilization rate, in each of the at least onecandidate node, of each type of resource in the resource provision isless than a corresponding utilization rate threshold; a type of resourcewith a current utilization rate less than the corresponding utilizationrate threshold is determined as an available resource of each of the atleast one candidate node; and the at least one candidate node matchedwith the already run Pod replica is determined according to the matchingof the demand resource and the available resource.

In the step 130, the already run Pod replica is scheduled to run on theat least one candidate node.

In some embodiments, the number of physical machines that need to beawakened by the at least one candidate node is determined according tothe predicted resource usage and the resource provision of the at leastone candidate node, wherein the number of physical machines is a minimumnumber capable of meeting the predicted resource usage.

In some embodiments, current resource usage can also be predictedaccording to a historical change of the resource usage. This can beachieved, for example, by the embodiments in FIG. 2 .

FIG. 2 illustrates a flow diagram of other embodiments of a schedulingmethod of containers according to the present disclosure.

As shown in FIG. 2 , the method can further comprise: step 210,acquiring a historical change; step 220, judging a current time period;and step 230, determining whether to increase a replica.

In the step 210, the historical change of the resource usage of thealready run Pod replica is acquired.

In some embodiments, average usage of each type of resource within aspecified time period is calculated according to the resource usage ofthe already run Pod replica at each time moment within the specifiedtime period; and according to the average usage, peak and valley timeperiods of the usage of each type of resource within the specified timeperiod are judged as the historical change.

For example, under the condition that a difference between the usage ofeach type of resource and the average usage of each type of resource isgreater than a peak threshold, the specified time period is determinedas a peak time period; and under the condition that a difference betweenthe average usage of each type of resource and the usage of each type ofresource is greater than a valley threshold, the specified time periodis determined as a valley time period.

In the step 220, according to the historical change, it is judgedwhether the to-be-scheduled Pod replica is currently in the peak orvalley time period of a usage.

In the step 230, it is determined whether to increase the number of theto-be-scheduled Pod replicas according to a result of the judgment.

In some embodiments, under the condition that it is in the peak timeperiod and a resource load of a single to-be-scheduled Pod replicaexceeds a load threshold, the number of the to-be-scheduled Pod replicasis increased.

In some embodiments, under the condition that it is in the valley timeperiod, the number of the to-be-scheduled Pod replicas is reduced, andan cleared physical machine is controlled to enter a sleep state.

In some embodiments, according to the historical change, resource usageof the to-be-scheduled Pod replica at a target time moment is predicted;and under the condition that the currently awakened physical machine isincapable of meeting the resource usage at the target time moment, anumber of physical machines which are capable of meeting the resourceusage at the target time moment are awakened to provide resources beforethe target time moment comes.

In some embodiments, according to the number of Pod replicas running ona corresponding physical machine of the at least one candidate node, thecorresponding physical machine is controlled to be in a high or non-highperformance mode.

For example, under the condition that there is no Pod replica running ona corresponding physical machine, the corresponding physical machine iscontrolled to be in a sleep mode; and under the condition that thenumber of Pod replicas running on corresponding physical machines isless than a replica threshold, the corresponding physical machine iscontrolled to be in a low performance mode.

FIG. 3 illustrates a schematic diagram of some embodiments of ascheduling apparatus of containers according to the present disclosure.

As shown in FIG. 3 , the scheduling apparatus can comprise a resourceusage evaluation module, historical change analysis module, a schedulingsystem, and a physical machine state control module.

In some embodiments, the resource usage evaluation module (which cancomprise a usage determination unit) is configured to evaluatestatistical data of usage of resources in a Pod. The statistical datacan provide a data basis for the historical change analysis module andthe scheduling system.

In some embodiments, the resource usage evaluation module can record andevaluate resource usage of Pods with a same Pod mode. The resource usagecan be represented by a 6-tuple. For example, the 6-tuple can comprise aCPU work instruction cycle count, memory usage, network IO (InputOutput) count, network IO traffic, storage IO count, and storage IOtotal.

In some embodiments, the resource usage can be a numerical value countedwithin a unit time t. For example, t can be 1 second, 1 minute, 1 hour,or the like.

In some embodiments, an actual work instruction cycle count of a CPU canbe used to accurately measure actual usage of the CPU by the Pod; thememory usage can be an accumulation value of memory usage per secondwithin the time period t; and the network IO and the storage IO can berepresented by count and total, for distinguishing cases of frequent IOand a large data amount of IO.

In some embodiments, the resource usage evaluation module, for the Podswith the same Pod mode, counts their resource usage, and calculates anaverage value thereof. The average value can also be divided by amaximum value that can be achieved by one physical machine to normalizethe average value, so that numerical values corresponding to the 6-tupleare unified into 6 numerical values between 0 and 1.

For example, a maximum value of a CPU that can be achieved by a physicalmachine is all cycle counts of all cores of the physical machine withint; and a maximum value of a memory that can be achieved by the physicalmachine is a total of all memories of the physical machine minus memoryusage of management and system programs.

A maximum value of the network IO count that can be achieved by thephysical machine is the most network IO count (updated when there is anew maximum value) that are historically performed by a system within t;a maximum value of the network IO traffic that can be achieved by thephysical machine is a maximum network transmission data amount that canbe achieved within t, which is calculated according to networkbandwidth; a maximum value of the storage IO count that can be achievedby the physical machine is the most storage IO count (updated when thereis a new maximum value) that are historically performed by the systemwithin t; and a maximum value of the storage IO total that can beachieved by the physical machine is a maximum transmission data amountthat can be achieved within t, which is calculated according to storagebandwidth.

A normalized numerical value can be obtained by dividing the actualresource usage of the Pods with the same Pod mode by the maximum value.Except for the IO count, a sum of the resource usage of the plurality ofPods on one same physical machine cannot be greater than a capabilityvalue of the physical machine.

In some embodiments, when specifications of a plurality of physicalmachines are not the same or processing capabilities thereof are notconsistent, the processing capabilities of the physical machines arealso normalized. For example, corresponding indexes of all physicalmachines must be divided by a value of a corresponding index of aphysical machine with a maximum processing capability, that is,normalized physical machine capability index=physical machine indexvalue/maximum physical machine index value.

In some embodiments, resource provision of a node is a normalized6-tuple, for measuring how many resources can be provided for the Pod bythe node.

In some embodiments, the historical change analysis module (which cancomprise a judgment unit) is configured to analyze a historical change(for example, a periodicity law) of usage of the Pod for resources suchas a CPU, memory, network, and IO, thereby providing a decision basisfor the scheduling system and the physical machine state control module.

In some embodiments, the usage of the Pod for the system resourceschanges in a wavy curve, which also generally shows a periodical changelaw. The change law of the system resource usage is generally caused byperiodicity of human social activities. For example, most people workduring the day and rest at night, causing systems interacting withpeople to be mostly busy during the day and idle at night. Therefore,better allocation of the resources can be realized by using the obtainedchange law.

In some embodiments, according to work, rest and activity laws ofpeople, a cycle of the change law can be divided into time periods ofday, week, month, year and the like.

In some embodiments, a law of resource usage of one day can be counted.For example, one time interval (for example, 1 minute) is taken as aunit time, and resource usage of Pods (Pod samples) with one same Podmode is counted within each unit time.

For example, abnormal data points can be removed within a time range(for example, 20 days) to obtain an average value of resource usage ofeach time period within one day; and time ranges of peak and valley timeperiods of the average value of the resource usage are determined.Resource usage in the peak time period is obviously higher than averageusage of each type of resource within one day; and resource usage in thevalley time period is obviously lower than the average usage of eachtype of resource within one day.

In some embodiments, a law of resource usage of one week can be counted.For example, a time period of a certain day is taken as a unit time tocount resource usage of Pods with one same Pod mode within each unittime. Within a time range (for example, 60 days), abnormal data pointsare removed to obtain an average value of resource usage of each daywithin one week. Time ranges of peak and valley time periods of theaverage value of the resource usage are determined.

In some embodiments, a law of resource usage of one month can becounted. For example, a time period of a certain day is taken as a unittime to count resource usage of Pods with one same Pod mode within eachunit time. Within a time range (for example, 90 days), abnormal datapoints are removed to obtain an average value of resource usage of eachday within one month. Time ranges of peak and valley time periods of theaverage value of the resource usage are determined.

In some embodiments, a law of resource usage of one year can be counted.For example, a time period of a certain day is taken as a unit time tocount resource usage of Pods with one same Pod mode within each unittime. Within a time range of one year, abnormal data points are removedto obtain an average value of resource usage of each day within oneyear. Time ranges of peak and valley time periods of the average valueof the resource usage are determined.

In some embodiments, through the four time spans of day, week, month andyear, the peak and valley time periods of the resource usage can beclearly analyzed, for guiding resource allocation.

In some embodiments, the scheduling system (which can comprise a nodematching unit and a scheduling unit) is responsible for dynamicscheduling of the system for Pod replicas when there is a Pod changesuch as increase, modification, and deletion. For example, dynamicscheduling mainly comprises scheduling when a Pod is created andscheduling when the Pod is modified and deleted. The scheduling systemcan comprise a scheduling strategy creation module and a dynamic changescheduling system.

In some embodiments, the scheduling strategy creation module can dividea scheduling strategy into three levels: a predicate strategy, apeculiarity strategy, and a set optimize strategy (or flex-growstrategy).

In some embodiments, the predicate strategy is used for filtering anodethat does not meet a set condition. The set condition can comprisemutually exclusive setting between a Pod replica and a node, forexample, a specified Pod replica cannot be deployed on a specified Node1.

In some embodiments, the peculiarity strategy is used for determining acandidate node set meeting conditions according to technicalcharacteristics demanded by the Pod replica after the filtering by thepredicate strategy. The technical characteristics can comprise thenumber of the Pod replicas, a high availability demand, an affinitydemand, an exclusivity demand, and the like.

In this way, performance and resource usage are not the onlyconsiderations for determining the at least one matched candidate node,and elements such as high availability, and high concurrency are alsointroduced, so that the scheduling effect is improved.

In some embodiments, the set optimize strategy is used for setoptimization and is a first way of third-level scheduling. For example,all to-be-scheduled Pod replicas can be taken as one Pod set forconsideration, and a strategy capable of forming optimal efficiencyafter the Pod set is scheduled to a candidate node set is selected forscheduling.

In some embodiments, optimal efficiency is that on the premise ofmeeting the set condition and the characteristic demand and under thecondition that part of emergency traffic processing space is reserved, aload is capable of meeting a business demand and minimizing the energyconsumption of the whole cluster.

For example, a workload can be allocated as far as possible to a smallnumber of physical machines, and the small number of physical machinesare capable of meeting resource and traffic demands, such that morephysical machines can sleep. When a physical machine to which theworkload has been already allocated has a less load, the physicalmachine enters an energy-saving mode, so as to have lowest energyconsumption while the business traffic demand can be met.

In some embodiments, the set optimization can comprise the case whereoperations for two Pods are considered at the same time, for example, itis needed to delete one Pod replica and then create one new Pod replica.If one of a plurality of Pod replicas needs to be deleted, a schedulingprogram is capable of selecting an optimal delectable Pod replica, suchthat a newly created Pod replica can achieve the optimal efficiency.

In some embodiments, when a Pod replica is created, a main flow of theset optimize method is as follows.

Step 1, according to a template file for creating the Pod replica,determining a mode of the Pod replica, a set condition for creating thePod replica, the number of the replicas and the like.

Step 2, excluding a node that does not meet the set condition. Forexample, a label is a label for identifying a Kubernetes object, and isattached to the object in a key-value manner. The label does not provideuniqueness and in fact, a same label is often used by many objects (forexample, Pods) to label a specific application. Some nodes can beexcluded by the label.

Step 3, dividing an allocatable candidate node set and distributionrelations between a plurality of replicas of the Pod according to thetechnical characteristics such as the number of the Pod replicas, thehigh availability demand, the affinity demand, and the exclusivitydemand.

For example, Pod replicas are allocated to at least two physical hostnodes to ensure high availability across the physical machines; or, thePod replicas are allocated to at least two racks of physical machines toensure high availability across the racks; or, the Pod replicas are allallocated onto physical hosts with high bandwidth networks to ensurehigh concurrent access to the networks.

Step 4, if a Pod replica which has run for a period of time and has asame Pod mode as the to-be-scheduled Pod replica exists, acquiring a6-tuple of actually used resource usage of recent resources of the Podmode according to the resource usage evaluation module.

Step 5, matching the resource usage 6-tuple of the Pod replica set witha remaining resource usage 6-tuple of candidate nodes in the candidatenode set, and calculating a matching strategy that enables the nodes toachieve 6 indexes balanced.

In some embodiments, if remaining resource usage in nodes on allphysical hosts in a non-sleep state can support the creation of a newPod replica, the Pod replica is scheduled to a corresponding nodeaccording to the balanced matching strategy. All to-be-scheduled Podreplicas are taken as one set for overall scheduling.

In some embodiments, several items with significantly higher indexes inthe resource usage of the Pod replica can be determined as high demandresources; and resources with maximum availability in the node areselected as valley resources (provision resources) to be matched withthe high demand resources. That is to say, indexes with large resourceidleness are filled as fully as possible, so that all 6 indexes realizebasically balanced allocation on the node.

In some embodiments, if some Pod replicas need to be scheduled to onesame Node, resource usage of the Pod replicas that need to be allocatedto the same Node is combined. For example, node resources are allocatedif they can meet the demand.

For example, FIGS. 4 a to 4 d are resource evaluation embodiments inwhich two Pod replicas are scheduled to one same Node.

FIG. 4 a illustrates a schematic diagram of some embodiments of noderesource distribution according to the present disclosure.

FIG. 4 a shows a normalized resource actual utilization rate andremaining allocable resource space of one schedulable node. In order toguarantee that the system is capable of carry business traffic under theconditions of traffic jitter, burst and the like, a reserved emergencyresource space is provided in the system. For example, proportions ofreserved emergency resource spaces of related indexes such as CPU andmemory can be different.

In some embodiments, a normalized average utilization rate of each typeof resource can be calculated, and according to the normalized averageutilization rate, valley resources with a lower utilization rate can bedetermined: memory, network IO traffic, and storage IO total.

FIG. 4 b illustrates a schematic diagram of some embodiments of Podreplica resource usage according to the present disclosure.

FIG. 4 b shows predicted resource usage that is predicted according tohistorical resource usage of already run Pod replicas with the same Podmode as the to-be-scheduled Pod replica. Normalized average usage ofeach type of resource can be calculated, and according to the normalizedaverage usage, high demand resources of the usage are determined:memory, network IO traffic, and storage IO total.

FIG. 4 c illustrates a schematic diagram of some embodiments of resourceusage of two Pod replicas according to the present disclosure.

FIG. 4 c shows predicted resource usage of the two Pod replicas, eachindex being twice that of a single Pod. High demand resources are:memory, network IO traffic, and storage IO total. In this case, the highdemand resource type and the valley resource type are just matched andthe number of the valley resources can meet the high demand resources,then the two Pod replicas can be scheduled onto the node.

FIG. 4 d illustrates a schematic diagram of some embodiments of ascheduling method of containers according to the present disclosure.

FIG. 4 d shows resource usage when two Pod replicas are allocated to aschedulable node. The case of a plurality of nodes is similar to that ofthe single node, and both need to ensure that the predicted resourceusage of the Pod can be met.

In some embodiments, if remaining resource usage in nodes on allphysical hosts in a non-sleep state is not enough to support thecreation of a new Pod replica, several sleeping physical hosts areawakened. Moreover, allocable resource usage of nodes on the awakenedphysical hosts and the original nodes is enabled to meet the resourceusage total demand of the mode for creating the Pod replica.

In some embodiments, if there is no already run Pod replicas with thesame Pod mode, the resource usage 6-tuple is predicted according to theresource demand (specification corresponding to each mirror image) inthe template document. For example, the more cores are demanded, thegreater the predicted CPU usage is. Then, according to the predictedresource usage, processing is performed according to the aboveembodiment.

In some embodiments, after all the Pod replicas are scheduled onto thenodes or all physical machines are awakened, no node capable of meetingthe resource demand exists, then the scheduling is finished.

In some embodiments, a set optimization model of the set optimizestrategy comprises: a target being that physical machines in thenon-sleep state are minimized; a constraint condition being that thecreation of all the Pod replicas meets the condition set in thetemplate; the creation of all the Pod replicas meeting the technicalcharacteristic demand; the created Pod replica being capable of meetinga business load; and resource usage of the Pod replica on each Node notexceeding a limit.

In some embodiments, the flex-grow strategy is used for flexibleallocation of the Pod replicas, which is a second alternative of thethird-level scheduling. For example, on the premise that constraintconditions of first two levels of scheduling are met, a part of Podreplicas are created first, and then the number of the Pod replicas isadjusted with reference to workloads of the Pod replicas.

In some embodiments, when a Pod is created, a main flow of the flex-growflexible scheduling method is as follows:

step 1, according to a template file for creating a Pod replica,determining a mode of the Pod replica, and determining a set conditionfor creating the Pod, and the number of the replicas.

Step 2, excluding a node that does not meet the set condition for thecreation. For example, certain nodes are excluded by a label.

Step 3, dividing an allocatable candidate node set and distributionrelations between a plurality of replicas of the Pod according to thetechnical characteristics such as the number of the Pod replicas, thehigh availability demand, the affinity demand, and the exclusivitydemand.

For example, Pod replicas are allocated onto at least two physical hostnodes to ensure high availability across the physical machines. Or thePod replicas are allocated onto at least two racks of physical machinesto ensure high availability across the racks. Or the Pod replicas areallocated onto physical hosts with high bandwidth networks to ensurehigh concurrent access to the networks.

Step 4, selecting a maximum number of replicas in two cases as thenumber of the Pod replicas to be created. For example, the two casescomprise half of the number in Pod replicas created demand, a minimumnumber of replicas meeting the technical characteristics.

Step 5, performing Pod scheduling according to the number of Podreplicas determined in the step 4, and minimizing the number of physicalmachines in the non-sleep state under the condition that the setcondition is met.

Step 6, increasing the number of Pod replicas when the resource usage ofthe Pod replicas is greater than an upper threshold. There are twooptions of a stride for the increase, in which one is that a unitincrease is performed according to a set number and the other is that aunit increase is performed according to a set proportion, for example,stride can be the half after the total number minus the existing number.When increasing the number, the target that physical machines in thenon-sleep state are minimized should also be met.

Step 7, decreasing the number of the Pod replicas when the resourceusage of the Pod replicas is less than a lower threshold. There are twooptions of a stride for the decrease, in which one is that a unitdecrease is performed according to a set number and the other is that aunit decrease is performed according to a set proportion, for example,the stride can be half of the existing number.

Step 8, repeating the steps 6 and 7, until the resource usage of the Podreplicas is between the upper and lower thresholds.

Step 9, if resources of the physical machines in the steps 6, 7 and 8are cleared, recycling the physical machines and enabling the physicalmachines to enter the sleep state, and if the resources of the physicalmachines are insufficient, awakening sleeping physical machines toincrease the resource provision.

In some embodiments, the dynamical change scheduling system performsscheduling using a flexible replica set that depends on the load stateof each current Pod replica. For example, the number of Pod replicas, 3,which is set by the user, can be taken as a standard value; when anoverall load of the Pod replicas is lower, the number of the Podreplicas is decreased under the condition that the constraint conditionsuch as high availability is met; and when the overall load of the Podreplicas is higher, the number of the Pod replicas is dynamicallyincreased to adapt to the demand.

In this way, the flexible replica set operation makes it possible toreduce the demand for a large number of computation resources in thecluster. Moreover, it can also enable part of physical machines in thecluster to enter the sleep state, to reduce the energy consumption ofthe cloud computing cluster. Since the electricity bill is one of maincontinuous expenses of the cloud computing platform, the running costcan be reduced.

A scheduling flow of the dynamic change scheduling system comprises:

step 1, analyzing a periodical law of resource usage of one Pod replicaset for providing a service. As mentioned above, a peak time period, avalley time period and an average value of the resource usage can beanalyzed according to four time spans of day, week, month and year.

For example, resource usage of one service is a sum of resource usage ofall Pod replicas it contains. The system needs to ensure that a serviceresource demand total can be supported regardless of how many Podreplicas there are in total.

Step 2, through a maximum value of resource usage of a single Podreplica, calculating the number of Pod replicas required by a maximumresource demand (traffic peak) in each time period; and calculating thenumber of Pod replicas required by a minimum resource demand (trafficvalley) in each time period.

Step 3, when the system is in the traffic peak, reserving enoughresources to support the scheduling of the Pod replicas.

Step 4, during the normal running of the system, allocating several Podreplicas capable of meeting task load processing according to an averageresource demand of the service.

Step 5, when the traffic peak comes, a resource demand increases, and aresource load of a single Pod replica exceeds a set threshold, startingthe flow of increasing the number of the Pod replicas.

Step 6, increasing the number of the Pod replicas. For example, if thereare not enough resources for allocation in the cluster, awakening thesleeping physical machines to increase the resource provision.

In some embodiments, a time TO taken for awakening and initializing aphysical machine is recorded; it is predicted that available physicalhost resources in the cluster at a time T will not meet the demandaccording to the periodical change of the resource usage; and an actionof awakening a physical host is started at a time T-T0.

Step 7, after the physical resource is in place, scheduling the Podreplicas and associating the Pod replicas with the correspondingservice.

Step 8, if the resource demand continues increasing, repeating the flowfrom the step 5 to the step 7, until the demand does not increase anymore.

Step 9, when the traffic demand decreases and it is analyzed accordingto the resource demand law that the resource demand will restore to thenormal state, gradually decreasing the number of Pod replicas; andcontrolling remaining Pod replicas to run with a high load and becapable of meeting the traffic demand. The Pod replicas can be scheduledas far as possible to a small number of physical hosts, to enable ancleared physical host to enter the sleep state.

Step 10, repeating the step 9 until the number of the Pod replicas isrestored to the number required during the normal running of the system.

Step 11, when the traffic valley comes, decreasing the number of the Podreplicas first, which is contrary to the flow of processing the trafficpeak; scheduling the Pod replicas as far as possible to a small numberof physical hosts; enabling an cleared physical host to enter the sleepstate. When the traffic increases, the number of the Pod replicas isincreased to a normal value.

In some embodiments, the physical machine state control module (whichcan comprise a control unit) controls different states of the physicalmachine, such as a sleep mode, an energy-saving mode, and a highperformance mode, according to a resource demand. For example, thephysical machine mode can be controlled by the running of a Pod samples.This can be achieved, for example, by the embodiments of FIG. 5 .

FIG. 5 illustrates a schematic diagram of other embodiments of ascheduling method of containers according to the present disclosure.

As shown in FIG. 5 , when a physical machine is cleared, that is, thereis no running Pod sample on it, it is judged whether recent businesstraffic will rebound. Under the condition that it will not rebound, thephysical host is enabled to enter the sleep mode to minimize the energyconsumption.

When only a small number of Pod samples (the number of which is lessthan the number threshold) are running on the physical machine and theactual resource usage is below the capability that can be provided bythe physical machine, the physical machine is enabled to enter theenergy-saving mode (low performance mode). In this way, the energyconsumption can be reduced while the resource usage is guaranteed.

When the physical machine is in the sleep mode and the system evaluatesthat more physical machine resources will be needed, a sleeping physicalmachine is awakened to enter the energy-saving mode, such that theawakened physical machine prepares for the resources and system runningenvironment.

When the physical machine is in the energy-saving mode and the actualresource usage increases, the physical machine in the energy-saving modecannot meet the performance demand, then the physical machine is enabledto enter the high performance mode.

When the physical machine is in the high performance mode and the actualresource usage decreases, the actual usage is far lower than thecapability of the physical machine, then the physical machine is enabledto enter the energy-saving mode.

In the above embodiments, through the scheduling of the Pod samples,part of the physical machines in the container cloud enter the sleepstate, thereby greatly reducing the energy consumption of the containercloud platform cluster and reducing the electricity cost.

In the above embodiments, by adopting the resource usage evaluationmethod based on the 6-tuple and the Pod mode, the possible resourceusage of the Pod replica is evaluated more accurately, thereby providinga more accurate basis for the scheduling of the Pod replica.

In the above embodiments, the periodical law analysis method for theresource usage provided on the container cloud, by adopting a higherperspective, can better judge the time for allocation and recovery ofthe Pod replicas and resources.

In the above embodiments, a three-level Pod creation schedulingmechanism is provided, such that the Pod scheduling is performed for thegoal of fewest physical machines, on the premise that various constraintconditions are met.

In the above embodiments, there is provided a Pod dynamic changescheduling mechanism. On the premise that the system load demand is met,the Pod is dynamically scheduled, for the goal of recycling resourcesand saving energy when the business traffic is in valley. When thebusiness traffic increases, the business demand can also be met byallocating the resources.

In the above embodiments, the resource usage of the to-be-scheduled Podreplica is predicted according to the priori data of the resource usageof the Pod replica of the same type as the to-be-scheduled Pod replica;on the basis, the scheduling strategy is generated in conjunction withthe number of resources which can be provided by each node. In this way,the resource usage of the system can be matched with the resourceprovision, to reduce the waste of the system resources.

FIG. 6 illustrates a block diagram of some embodiments of a schedulingapparatus of containers according to the present disclosure.

As shown in FIG. 6 , the scheduling apparatus 6 of containers comprisesa usage determination unit 61, a node matching unit 62, and a schedulingunit 63.

The usage determination unit 61 determines predicted resource usage of ato-be-scheduled Pod replica according to resource usage of an alreadyrun Pod replica, wherein types of the already run Pod replica and theto-be-scheduled Pod replica are the same.

The node matching unit 62 determines at least one candidate node matchedwith the already run Pod replica according to the predicted resourceusage and resource provision supported by candidate nodes.

In some embodiments, the node matching unit 62 determines the at leastone candidate node matched with the already run Pod replica according towhether the predicted resource usage of each type of resource in thepredicted resource usage is matched with provision of each type ofresource in the resource provision.

In some embodiments, the node matching unit 62 judges whether thepredicted resource usage of each type of resource in the predictedresource usages is greater than a corresponding usage threshold;determines a type of resource greater than the corresponding usagethreshold as a demand resource; judges whether a current utilizationrate, in each of the at least one candidate node, of each type ofresource in the resource provision is less than a correspondingutilization rate threshold; determines a type of resource with a currentutilization rate less than the corresponding utilization rate thresholdas an available resource of each of the at least one candidate node; anddetermines the at least one candidate node matched with the already runPod replica according to the matching of the demand resource and theavailable resource.

The scheduling unit 63 schedules the already run Pod replica to be runon the at least one candidate node.

In some embodiments, the scheduling unit 63 determines the number ofphysical machines that need to be awakened by the at least one candidatenode according to the predicted resource usage and the resourceprovision of the at least one candidate node, wherein the number ofphysical machines is a minimum number capable of meeting the predictedresource usage.

In some embodiments, the scheduling apparatus 6 further comprises ajudgment unit 64 configured to judge whether the to-be-scheduled Podreplica is currently in a peak or valley time period of a usageaccording to an acquired historical change of the resource usage of thealready run Pod replica, so as to determine whether to increase thenumber of the to-be-scheduled Pod replica according to a result of thejudgment.

In some embodiments, the judgment unit 64 calculates average usage ofeach type of resource in a specified time period according to theresource usage of the already run Pod replica at each time moment withinthe specified time period; and according to the average usage, judgesthe peak and valley time periods of the usage of each type of resourcewithin the specified time period as the historical change.

In some embodiments, the scheduling apparatus 6 further comprises acontrol unit 65 configured to estimate resource usage of theto-be-scheduled Pod replica at a target time moment according to thehistorical change; and under the condition that the currently awakenedphysical machine is incapable of meeting the resource usage at thetarget time moment, awaken a number of physical machines which arecapable of meeting the resource usage at the target time moment toprovide resources before the target time moment comes.

In some embodiments, the control unit 65, under the condition that it isin the peak time period and a resource load of a single to-be-scheduledPod replica exceeds a load threshold, increases the number of theto-be-scheduled Pod replica.

In some embodiments, the control unit 65, under the condition that it isin the valley time period, decreases the number of the to-be-scheduledPod replica and controls an cleared physical machine to enter a sleepstate.

In some embodiments, the control unit 65 is configured to control,according to the number of Pod replicas running on a correspondingphysical machine of the at least one candidate node, the correspondingphysical machine to be in a high or non-high performance mode.

In some embodiments, the control unit 65 controlling the correspondingphysical machine to be in a high or non-high performance mode comprises:under the condition that there is no Pod replica running on thecorresponding physical machine, controlling the corresponding physicalmachine to be in a sleep mode; and under the condition that the numberof Pod replicas running on the corresponding physical machine is lessthan a replica threshold, controlling the corresponding physical machineto be in a low performance mode.

In some embodiments, each of the at least one candidate node isdetermined according to at least one of the number of theto-be-scheduled Pod replica, a high availability demand, an affinitydemand, or an exclusivity demand.

In the above embodiments, the resource usage of the to-be-scheduled Podreplica is predicted according to the priori data of the resource usageof the Pod replica of the same type as the to-be-scheduled Pod replica;on the basis, the scheduling strategy is generated in conjunction withthe number of resources which can be provided by each node. In this way,the resource usage of the system can be matched with the resourceprovision, to reduce the waste of system resources.

FIG. 7 illustrates a block diagram of other embodiments of a schedulingapparatus of containers according to the present disclosure.

As shown in FIG. 7 , the scheduling apparatus 7 of containers of thisembodiment comprises: a memory 71 and a processor 72 coupled to thememory 71, the processor 72 being configured to perform, based oninstructions stored in the memory 71, the scheduling method ofcontainers according to any of the embodiments of the presentdisclosure.

The memory 71 can comprise, for example, a system memory, a fixednon-transitory storage medium, and the like. The system memory hasthereon stored, for example, an operating system, an application, a bootloader, a database, other programs, and the like.

FIG. 8 illustrates a block diagram of still other embodiments of ascheduling apparatus of containers according to the present disclosure.

As shown in FIG. 8 , the scheduling apparatus 8 of containers of thisembodiment comprises: a memory 810 and a processor 820 coupled to thememory 810, the processor 820 being configured to perform, based oninstructions stored in the memory 810, the scheduling method ofcontainers in any of the embodiments described above.

The memory 810 can comprise, for example, a system memory, a fixednon-transitory storage medium, and the like. The system memory hasthereon stored, for example, an operating system, an application, a bootloader, other programs, and the like.

The scheduling apparatus 8 of containers can further comprise aninput/output interface 830, a network interface 840, a storage interface850, and the like. These interfaces 830, 840, 850, and the memory 810and the processor 820 can be connected, for example, through a bus 860.The input/output interface 830 provides a connection interface forinput/output devices such as a display, a mouse, a keyboard, and a touchscreen. The network interface 840 provides a connection interface for avariety of networking devices. The storage interface 850 provides aconnection interface for external storage devices such as an SD card anda USB flash disk.

It should be appreciated by those skilled in the art that theembodiments of the present disclosure can be provided as a method,system, or computer program product. Accordingly, the present disclosurecan take a form of an entire hardware embodiment, an entire softwareembodiment or an embodiment combining software and hardware aspects.Furthermore, the present disclosure can take a form of a computerprogram product implemented on one or more computer-availablenon-transitory storage media (comprising, but not limited to, a diskmemory, CD-ROM, optical memory, and the like) having computer-usableprogram code embodied therein.

So far, the detailed description has been made according to the presentdisclosure. Some details well known in the art have not been describedin order to avoid obscuring the concepts of the present disclosure.Those skilled in the art can now fully appreciate how to implement thetechnical solutions disclosed herein, in view of the foregoingdescription.

The method and system of the present disclosure may be implemented in anumber of ways. The method and system of the present disclosure can beimplemented, for example, in software, hardware, firmware, or anycombination of software, hardware, and firmware. The above order for thesteps of the method is for illustration only, and the steps of themethod of the present disclosure are not limited to the orderspecifically described above unless specifically stated otherwise.Furthermore, in some embodiments, the present disclosure can also beimplemented as programs recorded in a recording medium, the programscomprising machine-readable instructions for implementing the methodaccording to the present disclosure. Therefore, the present disclosurealso covers the recording medium having thereon stored the programs forexecuting the method according to the present disclosure.

Although some specific embodiments of the present disclosure have beendescribed in detail by way of examples, it should be understood by thoseskilled in the art that the above examples are for illustration only andare not intended to limit the scope of the present disclosure. It shouldbe appreciated by those skilled in the art that modifications can bemade to the above embodiments without departing from the scope andspirit of the present disclosure. The scope of the present disclosure isdefined by the attached claims.

1. A scheduling method of containers, comprising: determining predictedresource usage of a to-be-scheduled Pod replica according to resourceusage of an already run Pod replica, wherein types of the already runPod replica and the to-be-scheduled Pod replica are the same;determining at least one candidate node matched with the already run Podreplica according to the predicted resource usage and resource provisionsupported by candidate nodes; and scheduling the already run Pod replicato be run on the at least one candidate node.
 2. The scheduling methodaccording to claim 1, wherein the determining at least one candidatenode matched with the already run Pod replica comprises: determining theat least one candidate node matched with the already run Pod replicaaccording to whether predicted resource usage of each type in thepredicted resource usage is matched with provision of each type ofresource in the resource provision.
 3. The scheduling method accordingto claim 2, wherein the determining at least one candidate node matchedthe already run Pod replica comprises: judging whether the predictedresource usage of each type of resource in the predicted resource usageis greater than a usage threshold corresponding to each type ofresource; determining a resource with predicted resource usage greaterthan a corresponding usage threshold as a demand resource; judgingwhether a current utilization rate, in each of the at least onecandidate node, of each type of resource in the resource provision isless than a utilization rate threshold corresponding to each type ofresource; determining a resource with a current utilization rate lessthan a corresponding utilization rate threshold as an available resourceof each of the at least one candidate node; and determining the at leastone candidate node matched with the already run Pod replica according tothe matching of the demand resource and the available resource.
 4. Thescheduling method according to claim 1, wherein the scheduling thealready run Pod replica to be run on the at least one candidate nodecomprises: determining a number of physical machines that need to beawakened by the at least one candidate node according to the predictedresource usage and the resource provision of the at least one candidatenode, wherein the number of the physical machines is a minimum numbermeeting the predicted resource usage.
 5. The scheduling method accordingto claim 1, further comprising: acquiring a historical change of theresource usage of the already run Pod replica; judging whether theto-be-scheduled Pod replica is currently in a peak or valley time periodof a usage according to the historical change; and determining whetherto increase a number of the to-be-scheduled Pod replica according to aresult of the judgment.
 6. The scheduling method according to claim 5,wherein the acquiring a historical change of the resource usage of thealready run Pod replica comprises: calculating average usage of eachtype of resource within a specified time period according to theresource usage of the already run Pod replica at each time moment withinthe specified time period; and determining the peak and valley timeperiods of the usage of each type of resource within the specified timeperiod as the historical change according to the average usage.
 7. Thescheduling method according to claim 5, further comprising: estimatingresource usage of the to-be-scheduled Pod replica at a target timemoment according to the historical change; and awakening, under thecondition that the currently awakened physical machine does not meet theresource usage at the target time moment, a number of physical machineswhich meeting the resource usage at the target time moment to provideresources before the target time moment comes.
 8. The scheduling methodaccording to claim 5, wherein the determining whether to increase anumber of the to-be-scheduled Pod replica according to a result of thejudgment comprises: increasing the number of the to-be-scheduled Podreplica, under the condition that it is in the peak time period and aresource load of a certain to-be-scheduled Pod replica exceeds a loadthreshold.
 9. The scheduling method according to claim 5, wherein thedetermining whether to increase a number of the to-be-scheduled Podreplica according to a result of the judgment comprises: decreasing thenumber of the to-be-scheduled Pod replica, and controlling an clearedphysical machine to enter a sleep state, under the condition that it isin the valley time period.
 10. The scheduling method according to claim1, further comprising: controlling a physical machine corresponding tothe at least one candidate node to be in a high or non-high performancemode, according to a number of Pod replicas running on the physicalmachine corresponding to the at least one candidate node.
 11. Thescheduling method according to claim 10, wherein the controlling acorresponding physical machine of the at least one candidate node to bein a high or non-high performance mode comprises: controlling thecorresponding physical machine to be in a sleep mode, under thecondition that there is no Pod replica running on the correspondingphysical machine; and controlling the corresponding physical machine tobe in a low performance mode, under the condition that the number of Podreplicas running on the corresponding physical machine is less than areplica threshold.
 12. The scheduling method according to claim 1,wherein each of the at least one candidate node is determined accordingto at least one of a number of the to-be-scheduled Pod replica, a highavailability demand, an affinity demand, or an exclusivity demand.13.-15. (canceled)
 16. A scheduling apparatus of containers, comprising:a memory; and a processor coupled to the memory, the processor beingconfigured to perform, based on instructions stored in the memory, thefollowing steps: determining predicted resource usage of ato-be-scheduled Pod replica according to resource usage of an alreadyrun Pod replica, wherein types of the already run Pod replica and theto-be-scheduled Pod replica are the same; determining at least onecandidate node matched with the already run Pod replica according to thepredicted resource usage and resource provision supported by candidatenodes; and scheduling the already run Pod replica to be run on the atleast one candidate node.
 17. A non-transitory computer-readable storagemedium having thereon stored a computer program which, when executed bya processor, implements the following steps: determining predictedresource usage of a to-be-scheduled Pod replica according to resourceusage of an already run Pod replica, wherein types of the already runPod replica and the to-be-scheduled Pod replica are the same;determining at least one candidate node matched with the already run Podreplica according to the predicted resource usage and resource provisionsupported by candidate nodes; and scheduling the already run Pod replicato be run on the at least one candidate node.
 18. The non-transitorycomputer-readable storage medium according to claim 17, wherein thecomputer program, when executed by the processor, implements thefollowing steps: determining the at least one candidate node matchedwith the already run Pod replica according to whether predicted resourceusage of each type of resource in the predicted resource usage ismatched with provision of each type of resource in the resourceprovision.
 19. The non-transitory computer-readable storage mediumaccording to claim 18, wherein the computer program, when executed bythe processor, implements the following steps: judging whether thepredicted resource usage of each type of resource in the predictedresource usage is greater than a usage threshold corresponding to eachtype of resource; determining a resource with predicted resource usagegreater than a corresponding usage threshold as a demand resource;judging whether a current utilization rate, in each of the at least onecandidate node, of each type of resource in the resource provision isless than a utilization rate threshold corresponding to each type ofresource; determining a resource with a current utilization rate lessthan a corresponding utilization rate threshold as an available resourceof each of the at least one candidate node; and determining the at leastone candidate node matched with the already run Pod replica according tothe matching of the demand resource and the available resource.
 20. Thenon-transitory computer-readable storage medium according to claim 17,wherein the computer program, when executed by the processor, implementsthe following steps: determining a number of physical machines that needto be awakened by the at least one candidate node according to thepredicted resource usage and the resource provision of the at least onecandidate node, wherein the number of the physical machines is a minimumnumber meeting the predicted resource usage.
 21. The non-transitorycomputer-readable storage medium according to claim 17, wherein thecomputer program, when executed by the processor, implements thefollowing steps: acquiring a historical change of the resource usage ofthe already run Pod replica; judging whether the to-be-scheduled Podreplica is currently in a peak or valley time period of a usageaccording to the historical change; and determining whether to increasea number of the to-be-scheduled Pod replica according to a result of thejudgment.
 22. The non-transitory computer-readable storage mediumaccording to claim 21, wherein the computer program, when executed bythe processor, implements the following steps: calculating average usageof each type of resource within a specified time period according to theresource usage of the already run Pod replica at each time moment withinthe specified time period; and determining the peak and valley timeperiods of the usage of each type of resource within the specified timeperiod as the historical change according to the average usage.
 23. Thenon-transitory computer-readable storage medium according to claim 21,wherein the computer program, when executed by the processor, implementsthe following steps: estimating resource usage of the to-be-scheduledPod replica at a target time moment according to the historical change;and awakening, under the condition that the currently awakened physicalmachine does not meet the resource usage at the target time moment, anumber of physical machines meeting the resource usage at the targettime moment to provide resources before the target time moment comes.